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This article provides an in-depth case study analysis of a pilot project 
organized by the section “Strategic Analysis” of the Belgian Federal 
Police. Using the Delphi method, which is a judgmental forecasting 
technique, a panel of experts was questioned about future developments of 
crime, based on their expertise in criminal or social trends. The results 
demonstrate how police authorities could implement judgmental 
forecasting methods like Delphi methodology for the an ticipation of future 
criminal trends, and how this technique, applied under specific conditions, 
can complement current crime analysis techniques. This article will not 
focus on criminal trends that were forecasted in the pilot project, but on 
the preconditions for using the Delphi method in criminal justice policy. 
Hence, this article could contribute to future applications of judgmental 
forecasting techniques by practitioners in both criminal justice systems 
and other policy domains. Key Words: Delphi Method, Forecasting, 
Criminal Justice Policy, Police Priorities, Crime Analysis. 


There is a variety of methods and techniques for the involvement of individual 
citizens, social organizations and experts in the decision making process (Landcom, 
2002; COSLA, 1998; Thomas, 1995). The involvement of various stakeholders, in both 
lay and professional roles, is important and is often said to enrich policy decisions 
(Slocum, 2005; Surowiecki, 2004). This article will focus on the participation of experts 
in criminal justice policy, and particularly on the use of judgmental forecasting methods 
(i.e., the Delphi method where experts are anonymously questioned in consecutive 
rounds) to map future criminal trends. We will discuss a pilot project implemented by the 
section Strategic Analysis of the Belgian Federal Police 1 in cooperation with the authors 
of this contribution (staff of the Public Management Institute and the Leuven Institute of 
Criminology, both at the Katholieke Universiteit Leuven). We will not so much focus on 
the results of the crime forecasting exercise but merely on the methodology that was used 
in this pilot project. 

The pilot project that is analysed in this case study is part of the development of 
the “National Security (Crime) Image 2006” realized by the Belgian Federal Police 
(Nationaal Politieel Veiligheidsbeeld / Image Policiere Nationale de Securite 2006), 
which is the result of a detailed analysis of current safety problems and the assessment of 
future developments of crime. Because the prognosis of future developments, not only in 
crime but also in general, is extremely difficult (Armstrong, 2001), the section Strategic 
Analysis decided to organize a pilot project to discuss this matter with leading experts in 


1 The commissioning unit of this pilot project was the section Strategic Analysis of the Belgian Federal 
Police. We would like to thank Martine Pattyn and her team for the successful cooperation and the 
indispensable help in the case study analysis. 
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criminal investigation and experts in other relevant domains (e.g., private companies, 
public agencies and universities). The use of the Delphi method as a judgmental 
forecasting technique can help to successfully anticipate (even less plausible) future 
societal phenomena. This, in turn, could be beneficial for the crime and safety policy. 

This article begins with an introduction of the Delphi method, after which a 
justification is given, why this specific forecasting method is more appropriate for the 
participation of experts and relevant stakeholders in the pilot project than other 
forecasting methods. Second, the use of the Delphi method in the pilot project is analysed 
with a focus on implementation, the achievement of objectives, group dynamics, the 
cooperation of the participants and the contribution of forecasting methods to the 
prognosis of future developments of crime. We will conclude by listing conditions for 
successful participation of experts, pitfalls and lessons for future applications of the 
Delphi method. 

This article’s contribution intends to be twofold. First, it tries to combine insights 
from criminology, public management and participatory methodology. Criminology is a 
relevant discipline, because the article concentrates on the assessment of future 
developments of crime. By the adoption of the pilot project in a broader, long term 
strategy, the focus is on public management. The article also tries to contribute to the 
methodological literature, because various judgmental forecasting techniques are 
discussed and compared. Second, the article intends to make a contribution to the Delphi 
literature. The current literature is mainly concentrated on ways Delphi can be organized, 
but does not often focus on good practices or preconditions to apply the Delphi method 
successfully. This case study emphasizes the importance of prescriptive literature about 
the application of Delphi. 


The Delphi Method in the Literature 


What is Delphi? 

Delphi as we know it is developed in the 1950s by Dalkey and Helmer, members 
of the RAND corporation (Woudenberg, 1991). It is a forecasting technique, fonnerly 
used in the military, that allows researchers to collect opinions among several 
independent experts on one specific topic, avoiding face to face discussion. It is an 
interesting instrument that can be used to reach consensus through structured consultation 
between a group of people who may have very different perspectives and fields of 
expertise. The method is particularly useful where there is little or no published 
information on the subject under consideration. Therefore it is a very interesting tool in 
forecasting (Armstrong, 2001; Coutorie, 1995). 

In practice, Delphi is an iterative process in which the participants are questioned 
individually using a written or digital list of relevant questions about a complex issue. In 
consecutive rounds the participants complete the same questionnaire several times. After 
each round they receive feedback on their answers, both from the organizing team and 
their fellow participating experts. Through different rounds, their opinions become more 
and more well-argued and it becomes clear whether or not consensus is possible (Strauss 
& Zeigler, 1975). The main purpose of such a process in several rounds is the 
accumulation of arguments for several alternative solutions and/or the reaching of 
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consensus between different experts. After three rounds it is usually clear whether 
consensus is possible. If not, the result is a mere accumulation of opinions and expertise, 
which is also very useful (Slocum, 2005). 

The Anonymity Rule 

Anonymity is an important characteristic of the Delphi technique. This refers to 
the fact that only the initiators of the Delphi know the identity of the participants, and 
thus none of the experts knows by whom the opinions they read and discuss in 
subsequent rounds are expressed. During the whole process the experts participate 
independently, without meeting in person. They only receive anonymous feedback from 
other participating experts who are asked (after each round) to give comments to the 
contributions of other participants (Slocum, 2005; Powell, 2003). There are at least two 
important reasons why the requirement of anonymity is essential in the Delphi method 
(Keeney, Hasson, & McKenna, 2006). A first reason is the reduction of the risk of 
“groupthink” mechanisms (Janis, 1982), referring to the fact that the opinions of people 
in an interactive discussion often tend to an artificial consensus. Second, anonymity also 
facilitates the main purpose of the Delphi technique, which is to obtain as many insights, 
ideas, and approaches as possible. The “anonymity rule” allows participants to freely give 
their opinion. It provides them with a certain “protection” against sarcasm and mockery 
that might occur when their response would be considered politically incorrect, socially 
unacceptable, awkward, etc. It also provides them with the opportunity to formulate a 
divergent (but perhaps a very interesting) opinion, opposite or different from the ideas of 
authorities in the field. An ongoing anonymity avoids the verbal dominance of those 
authorities, so every participant will be heard (Goodman, 1987). 

There are, however, two main pitfalls of anonymity. First, the lack of “personal 
accountability” for one’s statements might lead participants to jump to conclusions 
without thinking it over (Slocum, 2005). Second, there is the risk that some experts could 
refuse to participate because they will not receive recognition for their individual 
contributions. This highly depends on the specific characteristics of the participating 
experts. 

In some cases the process is concluded with one interactive meeting. The experts 
then meet in person to discuss the issue face-to-face and get the opportunity to do some 
networking, which can be an important incentive for participation: there is also 
“something in it for them” (Dick, 2000). Obviously, the requirement of anonymity will be 
challenged in this face-to-face meeting. It is, however, possible to continue the 
anonymous process by for example using the method of role-playing. Each expert has a 
specific role to play, specifically supporter or opponent of a specific opinion, and their 
task is to fonnulate arguments pro/contra this point of view on the basis of their expertise, 
without giving away their own opinion. This is a challenging but rewarding task, because 
each point of view is analysed in-depth. 
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Criticism on the Method 

The Delphi method received some criticism (see for example: Sackman, 1974; 
Ford, 1975; Goldschmidt, 1975; Hill & Fowles, 1975; Riggs, 1983; Bardecki, 1984; 
Rieger, 1986; Van Dijk, 1990; Rowe, Wright, & Bolger, 1991). These authors map at 
least four central disadvantages of the Delphi method. First, the technique is based on 
expert knowledge, professional experience, beliefs, opinions, feelings and expectations. It 
is not based on facts and therefore the validity and the reliability of the results could be 
questioned. Second, reaching consensus is neither always possible nor necessary. The 
accumulation of opinions can often be sufficient. A third reason is a more practical one. 
Participating in a Delphi is rather time-consuming and labour-intensive, increasing the 
risk of not finding enough participants or not being able to keep them motivated 
throughout the entire process. Fourth, the participants could experience the anonymity 
rule as frustrating, for example because there is no direct interaction between them and 
other experts. The decision to organize a face-to-face meeting at the end of the process 
can be a solution, but then a highly qualified facilitator would be essential, which is not 
always feasible in practice. 

Despite these possible disadvantages, the wide use and the renewed interest in this 
topic in the last few decades (e.g., Coutorie, 1995; Loo, 2002; Christie & Barela, 2005; 
Keeney et al., 2006) confirm that the Delphi method is still one of the best techniques for 
forecasting based on expert opinions. Furthermore, these critics often do not propose an 
alternative, but they only introduce some adjustments to the original method, for example 
the combination of Delphi with data collection techniques that are considered more valid 
or reliable (e.g., time series or cross-impact analysis), the omission of the anonymity rule 
or the addition of one or more face-to-face workshops. 

Why is the Delphi Method Appropriate for this Pilot Project? 

There is a variety of methods that can be used for the participation of experts in 
forecasting crime. An important question is why Delphi is the most appropriate for the 
pilot project of the Belgian Federal Police. In order to make a well-considered and 
substantiated selection of one specific method, we selected a number of criteria (listed in 
table 1), inspired by the models of Slocum (2005) and the FOR-LEARN online foresight 
guide (2005), on the basis of which several participatory methods of qualitative data 
collection involving experts were compared. For this pilot project there were five 
requirements. First, the method had to address as many objectives as possible (diagnosis, 
prognosis and forecasting), because both current and future safety problems had to be 
addressed. Second, the level of common knowledge about the topic, the necessary 
maturity of the participants, the complexity of the subject and the controversy about it 
had to be suitable for the pilot project where experts participated in a forecasting exercise 
about criminal trends. Third, the method had to be appropriate for the participation of 
experts. Fourth, the budget had to be acceptable for a pilot project organized by a 
governmental agency and thus paid with taxes. Finally, the necessary time to implement 
the method had to be rather short to fit the policy planning of the Belgian Federal Police. 
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Table 1. Criteria for the Selection of Qualitative Data Collection Methods Involving Experts 


Objectives 

Diagnosis (understanding the current situation) 

Prognosis (looking forward to what could happen) 

Forecasting (predicting future developments) 

X: objective is relevant 

XX: objective is very relevant 

Topic 

Level of: 





+ +/- 

- 


Knowledge 

A lot of common 

Not much common 



knowledge 

knowledge 


Maturity 

People already have an 

Subject is new 



opinion 



Complexity 

Highly complex or 

Not at all complex or 



technical 

technical 


Controversy 

Very controversial 

Not at all controversial 

Participants 

Laymen, experts, stakeholders, social organizations, etc. 

Budget 

1 = inexpensive 




2 = little expensive 



3 = expensive 




4 = very expensive 


Time 

Short, medium, long or variable term (= preparation included) 

AVAILABLE 





Subsequently a number of participatory methods were selected - on the basis of a 
literature review on expert and citizen participation (Loyens & Van de Walle, 2006) - 
that seemed to be, at face value, appropriate for the forecasting of crime, which was the 
main goal of this pilot project. The selected methods are listed below: 

• Back casting is a specific type of scenario analysis, starting from ideal or worst 
case scenarios of the future and then a “backward analysis” to the current 
situation with special attention to the sequence of critical events or trends. 

• Brain box or group decision room refers to a digital interactive process between 
several participants who all take part behind computers in one room. They can 
react to a central thesis or several questions in an anonymous way and with the 
supervision of a moderator. After the anonymous typing session, there is an oral 
discussion. After having been able to give their opinion anonymously, most 
participants are then willing to openly defend it. 

• Cross-impact analysis is a method to analyse the future in the light of other 
possible futures. It refers to the fact that participants make an assessment of the 
chance a certain future event will take place if certain other events will or will not 
take place. This method can be used in combination with other participatory 
techniques, for example the Delphi method. 

• Delphi is an iterative process, in which participants with a certain expertise are 
(anonymously) questioned in various rounds. 

• Dynamic mind mapping is a brainstorming method that can be used to quickly 
determine priorities. 
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• Expert panels refer to (semi-)permanent small groups of experts, specialised in a 
specific domain of expertise, debating about highly technological or complex 
policy issues. 

• Focus groups are intensive face-to-face workshops with a small group of 
stakeholders or experts, brought together for the exploration of a specific topic. 
Through in-depth discussion a large amount of ideas can be collected. 

• Interviews are face-to-face conversations with one stakeholder or expert. This 
one-to-one situation creates the opportunity to gather in-depth information, but it 
misses the chance for interaction with other stakeholders. 

• Scenario analysis aims to list several possible long term scenarios of current 
social developments, after which specific actions can be formulated in the light of 
one particular scenario. 

The selected methods were then compared on the basis of the five criteria (see 
table 2). This systematic comparison led to the conclusion that the Delphi method was the 
best option. There are four important reasons for that. First, with this method all 
objectives (First criterion) that are relevant in mapping future trends starting from the 
current situation - diagnosis, prognosis and forecasting - can be addressed. Second, the 
topic of forecasting crime (Second criterion) is suitable to be dealt with in a Delphi, 
because there is not much common knowledge about future developments of crime and it 
is an extremely complex issue that deals with new developments about which people do 
not have an opinion yet due to uncertainty and a lack of knowledge. Third, the Delphi 
method is particularly developed for expert participation (Third criterion), which was the 
main target group in the pilot project. Last but certainly not least, Delphi is a very flexible 
method in terms of necessary budget and time (criteria Fourth and Fifth). It can be used 
in a short or long tenn, with a small or high budget. In sum, the Delphi method was 
selected because it was most consistent with the five criteria. 
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Table 2. Comparing Different Forecasting Methods on the Basis of Five Criteria 


Method 

Objectives 

Topic 

Participants (target groups) 

Budget 

Time 


o 

> 

o 

•z 

o 

1/1 

K 

Prognosis 

Forecasting 

5 

o 

« 

r 

M 

O 

o 

w 

Maturity 

Complexity 

Controversy 




Back casting 


X 

X 

- 

- 

+ 

+/- 

Everyone 

1-3 

Medium term 

Brain box 

X 



- 

- 

+ 

+-/ 

Small group with specific expertise 

3 

Variable 

Cross-impact analysis 


XX 


- 

- 

+ 

+/- 

Experts 

1-3 

Medium term 

Delphi method 

X 

X 

X 

- 

- 

+ 

+/- 

Experts 

1-3 

Variable 

Dynamic mind mapping 

X 

XX 


- 

- 

+/- 

+/- 

Stakeholders 

1 

Short term 

Expert panel 


XX 


- 

- 

+ 

+/- 

Experts 

2 

Variable 

Focus group 

XX 



+/- 

- 

+/- 

+/- 

Stakeholders 

1 

Short term 

Interview 

X 

X 

X 

- 

- 

+/- 

+/- 

Everyone 

2 

Short term 

Scenario analysis 


XX 


- 

- 

+ 

+/- 

Everyone 

1-3 

Medium term 
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Case Study: Delphi in the Belgian Federal Police 

Before going into the details of the pilot project in the Belgian Federal Police, we 
will first briefly discuss the role of the authors in this case study analysis. Both Professor 
Jeroen Maesschalck and Kim Loyens work at the Leuven Institute of Criminology, where 
they combine criminology and public administration. Professor Geert Bouckaert from the 
Public Management Institute is specialised in public administration. For this pilot project, 
the authors were invited by the funding body to provide support to list and compare 
several methods of crime forecasting involving experts, on the one hand, and to select the 
most suitable method and implement it in the pilot project, on the other. This article 
provides an evaluation of the whole process. Despite our involvement in the pilot project, 
our attempt is to maintain objectivity. 

In the case study the application of the Delphi method in a pilot project of the 
Belgian Federal Police is analysed. This pilot project was carried out between 2006 and 
2007 by the section Strategic Analysis, which was then part of the Direction of 
Coordination and Functioning of the Federal Police ( Directie van de Coordinatie en de 
Werking van de Federate Politie / Direction de la coordination et du fonctionnement de 
la Police Federate). The section Strategic Analysis is responsible for the realisation of 
the National Security (Crime) Image (2006, [NationaalPolitieel Veiligheidsbeeld/Image 
Policiere Nationale de Securite]), in which the current and long term safety problems in 
Belgium are reported. This document is an important source of information, on the basis 
of which the “National Security Plan ( Nationaal Veiligheidsplan) 2008-2011”, which 
contains police priorities for the coming years, is developed. 

The “National Security (Crime) Image (2006)” is developed on the basis of two 
types of analysis. First, current crime and safety problems are analysed, using innovative 
methods, like risk analysis (focusing on threat, vulnerability and impact) and geographic 
profiling. This leads to a systematic overview of the nature, scale, seriousness and 
significance of all kinds of criminal phenomena (e.g., theft, corruption, terrorism, sexual 
offences, etc.), offender groups and different kinds of targets in Belgium. This is, 
however, but one aspect of the National Security (crime) Image (2006). The second one 
is the assessment or prognosis of future developments of crime, which is extremely 
difficult, but indispensable in the process of establishing police priorities. These data are 
normally collected through the analysis of several data bases and interviews with 
specialists within or outside the Police, but in 2006 the decision was made to gather data 
through a Delphi exercise among experts. 

The Selection of Experts 

One of the first questions in the pilot project was which target population would 
be the most appropriate to participate in the Delphi and would thus be able to make the 
best contribution to the forecasting exercise of future developments of crime. The “risk 
escalator” of Renn (2003) can be helpful to answer this question. According to Renn, the 
nature of a policy issue is the most important factor when detennining target groups for 
policy participation. When a problem is complex, the involvement of external experts is 
the best option. When dealing with uncertainty (i.e., problems experts cannot solve on 
their own) one has to broaden the group aimed at. Then all stakeholders and directly 
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affected groups should be involved, because they are familiar with the interests, values 
and uncertainties in their domain of expertise. According to Renn, the involvement of the 
general public is only necessary when dealing with an ambiguous policy issue. One 
example of the latter would be the topic of scientific developments (e.g., clone 
technology) that could lead to ethical questions about the acceptability of certain 
practices. In such cases ordinary citizens can make a value judgment and determine 
whether certain decisions are acceptable and/or desirable. 

In this specific case, complexity and uncertainty play an important role, because 
predicting future criminal trends is both complex and a matter of uncertainty. Therefore, 
the involvement of external experts and other stakeholders (e.g., police officers and 
magistrates in the field) is required. The involvement of “ordinary citizens” could also be 
useful when discussing policy decisions based on the expert analysis of future 
developments of crime. The opinion of individual citizens can for example be valuable in 
determining the social acceptability of certain political priorities in the criminal policy 
(Organisation for Economic Co-operation and Development [OECD], 2001). However, 
the commissioner of the pilot project had already decided not to involve regular citizens, 
but experts. 

Because the validity of the results in a Delphi highly depends upon the “quality” 
of the experts, this phase is a very crucial and delicate one (Landeta, 2006). Therefore, 
the research team conducted a scan of Delphi literature and other material in the field of 
expert participation. Based on this literature review, an extensive - but probably not 
exhaustive - list of relevant selection criteria was developed (Webby & O’Connor, 1996; 
Rowe & Wright, 1999; Loo, 2002; Slocum, 2005; the FOR-LEARN online foresight 
guide, 2005; Landeta, 2006). Two general recommendations seem to prevail when 
selecting experts: (a) there is not one recipe and (b) first make a profile. There is indeed 
“not one recipe” in selecting experts. The relevant selection criteria depend on objectives, 
complexity, necessary expertise, method, budget, etc. Therefore, it is important to make a 
specific profile of the “type of experts” you want to involve. A profile consists of two 
features: composition and balance. First, composition refers to the necessary expertise 
and characteristics of the participants. The main question is which domain-specific 
knowledge is needed to give a valuable contribution to the participatory project. Second, 
there should be a balance in the division of opinions and approaches of the participants to 
avoid the situation in which the discussion is being dominated by one specific group in 
(dis)favour of a specific stock of ideas. Based on these principles and recommendations 
in the literature a series of criteria was selected for this pilot project. Specifically, each 
expert had to fulfill six conditions before he or she could be selected: expertise, 
commitment, autonomy, independency, open-mindedness, and motivation (see table 3). 
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Table 3. Criteria for the Selection of Individual Experts 

• Expertise : The participants must be experts in one or several specific and relevant 
domains. Therefore experts belonging to the following groups could be selected: the Police 
(federal and local), the ministries (“Federal Public Service” in Belgium) of Home Affairs and 
Justice respectively, insurance companies (e.g. experts in car theft, burglary...), academics, 
IT staff (e.g. experts on trends in cyber criminality), ha nk sector, experts in 
forgery/drugs/hormones/etc., investigative journalists, etc. 

• Commitment'. The participants should be willing to invest time and energy in the 
project. 

• Autonomy. Experts must be able to speak autonomously, and not (only) as a 
representative of an organization. The main purpose of a Delphi is to collect the opinions of 
experts and not the viewpoints of the organizations they represent. 

• Independence : An expert is independent if the outcome of the Delphi will not affect 
him/her in a direct manner, so he/she can speak freely without fearing the consequences for 
his/her own position. 

• Open-mindedness: The participants must be open to feedback and willing to give 
account for (and sometimes even reconsider) their own opinions. 

• Motivation : Motivation is important to avoid experts giving up during the process. In 

part this can be influenced by the organizing team, for example by giving background 
information about the importance of the project, emphasizing the fact that they are “perfect 
for the job”, explaining “what is in it for the expert” and how their participation can have an 
impact on strategic decisions and police priorities, etc. _ 

After this first selection, the commissioning unit also tried to obtain a “balanced 
group of experts” where there was equilibrium not only between heterogeneity and 
homogeneity, but also between creativity and practicality on the part of the participating 
individuals. First, we will discuss the heterogeneity versus homogeneity dilemma. A 
heterogeneous composition leads to the enrichment of the results, because different 
viewpoints and alternative ideas are heard (e.g., domain of expertise, world orientation, 
sex, age, region, etc). Still, a certain degree of homogeneity is necessary to be able to 
start from a common basis. Second, there is also the creativity versus practicality 
dilemma. Because forecasting future developments is a creative process, the group of 
participants should consist of at least some creative and visionary individuals, who can 
think “out of the box.” Still, when detennining concrete actions—as a consequence of 
those creative ideas—you also need several practice-oriented experts, who can translate 
long-term goals and strategic decisions into short-term actions. Their “practical 
awareness” can be very valuable. 

A final issue in the selection of participants is the number of experts. This mainly 
depends on the situation, topic, method, objectives, budget, etc. Armstrong (2001) 
recommends that there should be at least five participating experts in a Delphi. When 
there is also an oral discussion at the end of the process, he considers 20 participants the 
absolute maximum. In the pilot project 20 internal and 27 external experts were 
contacted, and about 30 to 60 percent (depending on the specific round in the Delphi) 
agreed to participate. For the internal police experts there was a lower participation level 
in the written round (unlike the external experts they only participated in one of the two 
written rounds) than in the oral round (respectively six versus 12 out of 20). For the 
external experts the participation in the first written round was quite high, but dropped 
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drastically in the second written round (respectively 14 versus three out of 27), to rise 
again in the oral round at the end (nine out of 27). 

The low participation level in some of the written rounds is most likely due to the 
fact that the Delphi started in the summer (in 2006). While of course some people are on 
holidays in this period of the year, this is often also considered a convenient time to 
finalize a publication or to get some other work done for which, otherwise, there is no 
time. Still, the organizing team was satisfied with the response. Arguably, the 
participation rate is not only highly dependent on the nature of the topic, but also of the 
type of organization. Because “crime” is a hot topic and the section Strategic Analysis of 
the Police is a “high profile” organization, several experts considered it their duty to 
participate, especially those in the public sector where there was hardly any drop-out. In 
the academic world, however, the response was rather low. The latter could be explained 
by a very busy time schedule and the lack of immediate gain in academic terms. The 
intensive efforts of the Delphi participation would not lead to a publication or any form 
of personal recognition because the experts would contribute anonymously. 

The Organisation of the Delphi 

After the selection of experts, the actual Delphi was organized between August 
2006 and January 2007. The decision was made to work with two different groups of 
experts. The first group would do the preparatory work for the actual Delphi by the 
second group. Because the pilot project was a forecasting exercise the results could be 
rather diffuse and incoherent if there would be no clear selection of topics. Hence, it 
seemed necessary to organize a preparatory phase, in which a small number of (mainly 
internal police) experts made a selection of issues that would be treated in the actual 
Delphi. In cases where the Delphi method is used in a forecasting exercise this could be 
an interesting innovation leading to a more focused and profound discussion about a 
small number of central issues. Table 4 (see below) gives an overview of the numbers of 
experts that were contacted in the preparatory phase and the actual Delphi, and the 
numbers of experts that really participated in each round. 

Table 4. Experts in the Preparatory Round and Actual Delphi 


Phase in Project 

Internal police experts 

External experts 

Preparatory phase (contacted) 

8 

4 

Preparatory phase (participation) 

3 

2 

Actual Delphi (contacted) 

±20 

±27 

Actual Delphi 1 st round (participation) 

/ 

14 

Actual Delphi 2 nd round (participation) 

6 

3 

Actual Delphi participation face-to-face round 

12 

9 


Preparatory workshops. The main purpose of the preparatory work was to 
provide the necessary material for the discussion about future criminal developments. 
The experts within these preparatory workshops were asked to identify social trends that 
could occur or further develop in the period between 2008 and 2011 (this medium term 
perspective corresponds to the period covered by the “National Security Plan” to which 
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this forecasting exercise hopes to contribute). In order to keep the project manageable and 
focused, a small number of social trends were selected that could become driving forces 
for future delinquency and could thus have an impact on the future development of crime 
(e.g., globalisation, multiculturalism, technological developments, etc). Two workshops 
were organized to brainstonn with a small group of experts, in various rounds, about 
possible driving forces and organize them in different categories (e.g., social, economic, 
political, etc.). Because this activity required both knowledge of criminal phenomena and 
creativity, the decision was made to select several specialists within the Police, but also 
some external experts, able to “think out of the box.” 

The result of these interactive meetings was, on the one hand, a list of seven 
driving forces that could have an impact on future crime and safety problems - 
particularly globalisation, migration, individualism, polarisation, economical 
transformations, aging, technological revolution, and changed governmental tasks - and, 
on the other, a very preliminary identification of criminal trends linked to these driving 
forces. These sessions together with a limited literature review by the commissioning unit 
offered the necessary basic information that could then be further elaborated in the actual 
Delphi. 


The actual Delphi. The actual Delphi consisted of three rounds. In round one and 
two the participating experts responded individually to a questionnaire (through e-mail). 
They met in person in round three, which was a face-to-face workshop. 

Round one. The questionnaire for the first round was sent to the external experts 
through e-mail. For each of the seven driving forces the participants were asked to 
provide a list of new social and criminal trends that could appear in the following five 
years (until 2011) as a consequence (at least partly) of these driving forces. The experts 
were also asked to give a brief description of each criminal trend: type of crime, expected 
location(s), target(s), tactic(s) and profile of offender(s). After completing the 
questionnaire individually and anonymously, they returned it to the organizing team. 
They analysed the answers of all experts, which led to an anonymous and structured 
document of 50 pages including all the written contributions about criminal trends for 
each driving force. 

Round two. In the second round, this summary was sent to both internal and 
external experts. Each external expert received feedback from the organizing team. This 
included two types of information: (a), requests for more explanation or argumentation 
about their own contribution (if necessary) and (b). confrontation with possible 
contradicting answers of different experts. The experts were asked to respond to the 
feedback and make an overall evaluation of the anonymous answers of all the 
participants, including their own. Although the experts would eventually meet, it was 
essential that the contributions remained anonymous. The results of the second round 
were then analysed by the commissioning unit. 

Round three. The third round - which consisted of two face-to-face workshops 
(one for the internal and one for the external experts) - started with a presentation of the 
organizing team where each driving force was explained and linked to criminal trends, 
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followed by an intensive debate by the experts who were asked to respond to the results 
of the Delphi. 


An Evaluation of the Pilot Project 

This section provides an overall evaluation of the application of the Delphi 
method in the pilot project. An attempt is made to identify the most important conditions 
for successful expert participation, but also pitfalls and lessons for future applications of 
the Delphi method. First, the importance of a preparatory phase, which could be an 
innovation to the Delphi method, is emphasized. Second, several implementation 
challenges are listed. We conclude with an overview of the most important contributions 
of the Delphi method. 

The Importance of a Preparatory Phase 

The preparatory work of the first group of experts was considered an essential 
part of the pilot project. The experts managed to provide a useful contribution to the 
actual Delphi, because the seven driving forces and the associated criminal trends that 
they identified really shaped the pattern of the next phase. The general list of driving 
forces was used to clarify the desirable direction of the expert contributions in the actual 
Delphi, while it offered a sufficiently broad scope of very diverse issues. In the Delphi 
literature the relevance of a preparatory phase is hardly discussed. Based on the 
experiences in the pilot project of the Belgian Federal Police the conclusion can be made 
that preparatory workshops with mainly internal experts can improve the quality of the 
actual Delphi, mainly because it could lead to more in-depth discussion by the external 
experts since they are confronted with specific topics that were already pre-analysed. 

Implementation Challenges in the Pilot Project 

In general, the section Strategic Analysis of the Belgian Federal Police considered 
the Delphi project a successful exercise, especially with respect to the results. There 
were, however, some difficulties in the implementation process. At least four challenging 
implementation issues can be distinguished. 

First, participation is voluntarily and entirely based on the goodwill of individual 
experts. For the organizer, it can be a hard and sometimes disappointing task to motivate 
participants. Because gifts and rewards were not considered an option (as a consequence 
of both ethical and budgetary reasons), a number of arguments were used in attempts to 
convince individual experts, going from “this project deals with a hot topic” and 
“participation could broaden your occupational network”, to “it is your duty as a public 
servant to participate.” The latter argument seemed to be quite convincing for members 
of governmental organizations that were contacted. It appears that these actors 
acknowledged their social responsibility (especially because the request came from the 
Police) and, thus, felt an intrinsic obligation to cooperate. A more difficult group to 
persuade were the academics, who gave reasons like overbooked agendas and urgent 
deadlines. In addition to the bad timing of the Delphi and the short deadlines that were 
imposed, there is another possible explanation for this. Contributions to the Delphi would 



1490 


The Qualitative Report November 2011 


be considered anonymous, implying that the names of the experts would only appear in a 
general list of participants in the final publication of the results. The lack of personal 
recognition for individual contributions could, especially for academics, lead to a 
negative answer on the question “What is in it for me?” Nevertheless, referring to the 
upcoming face-to-face workshop at the end, which could be a very good opportunity to 
do some networking, was convincing for some academics. This confirms the fact that, for 
the organizer, it is extremely important to build in several non-financial “rewards” (e.g., 
mentioning the names of the experts when presenting the final document to the 
authorities, providing opportunities to network, etc), thus giving participants the feeling 
they will receive the recognition they deserve. 

The second implementation challenge can be derived from the previous one. Due 
to the voluntary nature of the Delphi procedure, and the fact that cooperation is based on 
the goodwill of individual participants, the contributions of various experts differed to a 
great extent in terms of length, quality, depth, and thoroughness. Some reactions were 
rather minimalist, while others were real “pieces of art”, according to one of the 
commissioners. There were contributions of no more than two lines, but also elaborated 
texts that were compiled through the joint effort of several members of a working group 
that some of the participating experts assembled in their own organization. The experts’ 
creativity, inspiration, motivation and goodwill are, as stated above, important 
preconditions for a successful Delphi. The specific theme dealt with was, however, also 
an important determinant. Some topics, like for example “migration” and “technological 
revolution”, were elaborated to a higher extent than “economy” and “aging.” As for 
aging, several experts stated that the most drastic effects would only occur in the year 
2030, which is outside the scope of this forecasting exercise. This could, at least partly, 
explain why the issue has not been elaborated as extensively as the other ones. Migration, 
on the other hand, might have been discussed more extensively, because of its political 
sensitivity and visibility. It is unclear why “technological revolution” has been discussed 
more, and “economy” less. 

The third implementation challenge concerns discrepancies between the first and 
the second written round. Not only the number of external experts decreased drastically 
in the second round (from 14 to three), but also the quality and thoroughness of the 
contributions diminished. As for the latter, the experts in the second round did make a 
few adjustments to their own contributions, and came up with some bright ideas that 
occurred by way of association, but only to a rather limited extent. No participant dared 
to formulate feedback towards other experts or arguments for/against a particular 
statement. This could be explained by an understandable reservation to criticize other 
expert opinions (although this argument can be contested by referring to the anonymous 
nature of the Delphi) or the lack of clear guidelines about the relevance and usefulness of 
the second round. Some experts could have had the idea that all had already been said 
and done in the first round. Why, then, formulate one’s opinion once more? Nevertheless, 
the ultimate purpose of the Delphi method is the accumulation of elaborated and well- 
argued opinions or ideas, which could be achieved in an iterative process of several 
rounds in which experts formulate answers to the proposed feedback (Strauss & Zeigler, 
1975). The drastic decrease of experts in the second round could also show a decline of 
motivation for further participation in the project. This is, however, inconsistent with the 
observation that in the final face-to-face round no less than nine external experts in 



Kim Loyens, Jeroen Maesschalck, and Geert Bouckaert 


1491 


workshop one (three times more than in the second round) and 12 police experts in 
workshop two (two times more than in the written round) were present. 

A final implementation issue concerns timing, which was one of the main 
problems in the pilot project. Because of a number of delays the postulated timing could 
not be respected. Delay can be caused by several factors, including personnel shortage, 
workload problems, lack of resources, etc. However, in this particular pilot project it was 
presumably caused by the fact that many of the participating experts did not respect the 
deadlines. In the first round, only one of them responded in time. The others only 
responded after having been reminded. The members of the commissioning unit mention 
three possible explanations. First, as mentioned above, participation is voluntary, so the 
organizing team could not force the experts to respect the deadline. Second, the target 
group was a high profile group of experts with a busy schedule. Third, the timing was not 
ideally chosen. While the summer holidays appeared ideal, it turned out that many of the 
respondents had different priorities in that period. To deal with the timing issues, the 
commissioning unit responded in two ways. On the one hand, the remaining experts who 
did not yet complete the questionnaire were interviewed leading to a higher workload for 
the organizing team. On the other, the deadline was postponed in the first round. 
However, in the second round, late submissions were disregarded. This could probably 
explain a great deal of the drastic drop-out in the second round. In the final face-to-face 
workshop, a larger group of experts was attracted once more. 

Main Contributions of the Delphi Method in the Pilot Project 

Notwithstanding the challenging implementation issues mentioned above the pilot 
project was still considered satisfactory. Four major contributions of the Delphi method 
in this specific case are worth mentioning. First, the most important contribution of the 
Delphi was, according to the commissioning unit, the collection of various ideas and 
opinions concerning one particular topic, coming from different perspectives. Although 
the ideas themselves were not always that innovative as such (at least not from the 
viewpoint of an expert), the result was innovative because expertise from different 
domains was centralized and applied to the topic of criminal trends. 

A second contribution is the seemingly less successful second round of the 
Delphi. While the first round was of the utmost importance for the accumulation of 
knowledge and expertise, the second round (despite the weaknesses mentioned above) 
was also considered essential. The main reason for that was that this round turned out to 
be a necessary preparatory phase for the face-to-face round. By first thinking and writing 
about the topic independently (in round one), and then reflecting and (for some) 
commenting on other experts’ texts (in round two), the participants were ready to discuss 
the topic in a focused and in-depth manner during the group interaction (in round three). 
Without the second round, where they could read and think about other participants’ 
opinions, the risk of the face-to-face workshop ending up in an overly broad, vague and 
empty discussion would have been much higher. 

A third contribution is the avoidance of narrow-mindedness by using the Delphi 
method. By looking at a particular issue from various perspectives—in this case the 
different domains of expertise of the participants—the scope of analysis is broadened 
extensively. 
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A final contribution of the Delphi method, is an unintended side effect. Because 
the face-to-face workshop was, by most participants, considered extremely interesting 
and valuable, the idea was raised to continue the activities of this forum of experts in a 
new deliberative platfonn which would regularly meet (for example once a year). 

Conclusions and Recommendations 

This article attempted to demonstrate how the Delphi method can be used in the 
participation of experts in the forecasting of criminal trends. In our opinion, this case 
study analysis could be relevant for both practitioners and researchers. 

As for practitioners, at least four necessary conditions of success for future 
applications of the Delphi method can be formulated. First, the most crucial precondition 
of a successful Delphi is, obviously, the quality and reputation of the participating 
experts. Hence, a well-considered selection phase—with a clear profile, an appropriate 
composition and the ideal number of participants—is of the utmost importance. The 
experts’ reputation is particularly important when communicating the results. Still their 
opinions should be treated with caution and preferably verified. This verification could be 
organized in at least two ways: (a) by conducting an additional literature analysis or (b) 
by attracting more than one expert in a specific domain of expertise so as to allow the 
experts to correct or complement each other’s contributions in the subsequent rounds. 

A second condition for success is the motivation of experts, which can be 
increased by providing several non-financial “rewards” such as the mentioning of the 
experts’ names when presenting the final document to the authorities or providing 
opportunities to network. The organization of a face-to-face workshop in the end was a 
crucial motivator in this particular project, which became clear in the increase of 
participation in round three (the workshop), compared to round two (written round). 

Third, clear communication of guidelines and expectations in the several Delphi 
rounds could improve the quality of the contributions and perhaps diminish the 
significant differences between the various texts in terms of length, depth and 
profoundness. The preparatory work of the first group of experts helped to avoid this 
problem, because they selected a small number of well-defined topics that would be 
treated by the second group of experts in the actual Delphi. Due to the voluntary nature of 
the project, one can, however, not enforce (and maybe one should not expect) high 
quality and professional contributions. Appealing to the experts’ sense of duty could be a 
convincing argument for some participants. Specifically for the second round, the 
commissioning unit should also determine the exact purpose of this additional reading 
and writing exercise. Is it the accumulation of knowledge and expertise or rather an 
attempt to reach consensus? This should also be communicated to the participants, 
because it certainly has an impact on their approach. 

Fourth, the timing is an essential condition of success. Not only the period of the 
year should be selected carefully (most suitable for the target group), but also the 
maximum duration of each round (not too long and not too short) should be detennined 
and maintained. It could be necessary to build in some back-up time after the deadline 
has passed (obviously concealed for the participants), during which the “disobedient” 
experts can be urged to still contribute. For those who are systematically delayed one 
could conduct an interview as an ultimate remedy. 
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It is important to add a final remark about an essential characteristic of the Delphi 
method: the anonymity of contributions. The literature suggests that the anonymity of the 
Delphi method creates some interesting benefits, like the decrease of the risk of “group 
think” mechanisms and social desirability bias. These were confirmed in the pilot project, 
but it became clear that the “anonymity rule” had one serious disadvantage. The 
academics were probably less motivated because their contribution would not lead to 
personal recognition nor to authorship of a publication. This should be kept in mind when 
organizing a Delphi. 

We conclude with one important suggestion for future research. The case study in 
the Belgian Federal Police suggests that it can be interesting to evaluate initiatives in 
which Delphi or, generally speaking, judgmental forecasting methods are applied. It 
would be particularly interesting to conduct further case study research, in which not only 
general rules of thumb are formulated, as is the case in most articles about the Delphi 
method, but also the specific conditions are analysed under which a participatory method 
in a particular project can be successful. 
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